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1.  Abstract: 

Background:  Specialized  Small  Target  Motion  Detector  Neurons  (STMDs)  in  the  optic  lobes  of  the 
insect  brain  respond  strongly  to  moving  objects  even  when  these  are  smaller  than  the  nominal 
resolution  limit  of  the  eye.  Many  STMDs  also  respond  robustly  to  small  targets  against  complex 
stationary  or  moving  backgrounds.  In  the  last  decade  the  Principal  Investigators  on  this  project  have 
established  the  underlying  neural  machinery  of  STMD  system  as  an  important  new  model  system 
for  computational  neuroscience  and  for  bio-inspired  models  of  target  tracking.  With  sponsorship 
from  the  US  Air  Force  Office  of  Scientific  Research  (Contracts  F49620-01-C-0030, 
FA9550-04-1  -0283,  and  Grants  FA9550-09-1-01 16  and  FA2386-10-1-41 14),  our  research  has 
allowed  us  to  characterize  and  model  key  physiological  properties  that  underlie  the  impressive 
visual  abilities  of  insects  such  as  predatory  dragonflies  to  discriminate  and  track  small  moving 
targets  against  cluttered  backgrounds,  despite  poor  optical  resolution  (low  pixel  count).  In  the 
process,  we  have  established  a  foundational  literature  on  this  topic. 

Aims  &  Experiments:  In  this  grant,  we  hypothesized  that  these  properties  required  a  complex 
mechanism  to  avoid  breakthrough  responses  by  background  features,  yet  to  adequately  amplify  the 
weak  signal  of  tiny  targets.  We  combined  electrophysiological  recording  from  an  identified  dragonfly 
STMD  neuron,  CSTMD1,  with  computational  modeling  of  its  input  pathway  (elementary  small  target 
motion  detectors,  ESTMDs).  Our  data  show  evidence  for  responses  along  long  trajectories  being 
strongly  facilitated  by  a  mechanism  that  builds  up  slowly  over  several  hundred  milliseconds.  This 
allows  the  neurons  to  give  sustained  responses  to  continuous  target  motion,  thus  providing  a 
possible  explanation  for  their  extraordinary  contrast  sensitivity.  Computational  modeling  shows  that 
this  cannot  be  accounted  for  by  simple  emergent  properties  of  existing  bio-inspired  motion  detector 
systems.  We  report  here  on  progress  towards  understanding  the  complex  operations  within  the 
STMD  receptive  field  that  explain  their  extraordinary  selectivity.  We  further  developed  a  discrete 
time,  digital  implementation  of  a  complete  closed-loop  target  pursuit  system  with  an  ESTMD  front 
end  which  can  be  used  to  evaluate  the  performance  of  the  ESTMD  model  as  a  front  end  for 
simulation  of  target  pursuit  in  visual  clutter. 

Key  findings:  We  document  data  exploring  two  further  discoveries  based  on  recordings  from  the 
dragonfly  CSTMD1  neuron:  (1)  Evidence  that  STMDs  use  a  non-linear  interaction  (as  previously 
hypothesized)  between  contrast  decrements  (OFF  features)  and  increments  (ON  features).  This  is 
the  first  time  convincing  evidence  has  been  obtained  for  such  an  interaction  in  insect  vision,  and 
suggests  that  STMDs  use  a  fundamentally  different  underlying  mechanism  for  feature  selectivity 
compared  with  other  biological  motion  detectors  (such  as  the  famous  Reichardt  EMD  model).  (2) 
We  further  developed  several  variants  of  computational  models  that  capture  the  key  properties  of 
direction-selective  SF-STMD  neurons  by  cascading  partially  rectified  ESTMD  stages  with  FIR-type 
EMDs,  and  vice  versa.  We  then  examined  the  key  response  tuning  and  predictions  of  these  second 
order  systems  for  simple  stimuli.  We  found  that  the  combination  of  an  ESTMD  front-end  with  a  2nd 
order  EMD  is  able  to  combine  both  the  direction  selectivity  of  classical  EMDs  with  the  feature 
selectivity  of  the  ESTMD  model.  Moreover,  this  system  explains  the  stronger  dependence  on 
feature  contrast  in  insect  STMDs  compared  with  their  (EMD  based)  wide-field  motion  sensitive 
counterparts.  Finally,  this  model  captures  the  sharper  response  tuning  for  either  size  or  velocity  of 
targets  observed  in  insect  STMD  neurons  compared  with  the  predictions  of  either  EMD  or  ESTMD 
models  alone. 
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2.  Introduction: 

2.1  Background  and  Objectives 

Many  animals  visualize  and  track  small  targets  at  large  distances  -  be  they  prey,  approaching 
predators  or  conspecifics.  Insects  are  an  excellent  model  system  for  investigating  the  neural 
mechanisms  that  have  evolved  for  this  challenging  task.  Specialized  Small  Target  Motion  Detector 
Neurons  (STMDs)  in  the  optic  lobes  of  the  insect  brain  respond  strongly  to  targets  that  are  smaller 
than  the  nominal  resolution  limit  of  the  eye.  Such  a  feature  must  be  blurred  to  a  very  low  contrast  in 
the  retinal  image,  so  its  detection  and  tracking  requires  enormous  contrast  gain.  Many  STMDs  also 
respond  robustly  to  small  targets  against  complex  stationary  or  moving  backgrounds,  which  further 
complicates  the  task  of  detecting  the  feature,  since  this  background  clutter  both  reduces  absolute 
contrast  and  adds  conflicting  cues  from  other  contrasting  features  of  the  background  image. 

In  this  grant,  we  originally  hypothesized  that  the  extraordinary  abilities  of  STMDs  stem  from  a 
complex  mechanism  to  avoid  breakthrough  responses  by  background  features,  yet  to  adequately 
amplify  the  weak  (blurred)  signal  produced  by  tiny  targets.  The  primary  objective  was  to  improve 
understanding  of  the  computational  principles  that  underlie  the  response  characteristics  of 
wide-field  Small  Target  Motion  Detector  neurons  in  the  insect  visual  system  (STMDs).  A  secondary 
objective  was  to  achieve  a  more  in-depth  empirical  characterization  of  the  cellular  (neuron) 
response,  particularly  with  regard  to  details  that  bear  on  the  limits  of  performance  as  small  moving 
target  detectors,  and  on  unresolved  questions  regarding  the  underlying  computational  mechanisms 
of  the  biological  STMD  system.  Our  original  Aims  were  as  follows: 

AIM  1.  Rectifying  Transient  Cells:  The  goals  were  to  determine  whether  or  not  neurons  of  this 
class  in  fact  lie  on  the  neural  pathway  leading  to  STMDs;  to  obtain  a  more  detailed 
characterization  of  physiology  and  anatomy,  and  to  perform  modeling  and  analysis  that 
accounts  for  details  of  the  physiology  and  the  putative  role  of  RTCs  in  STMD  processing  not 
yet  addressed. 

AIM  2.  Nonlinear  Facilitation  for  Long-Range  Spatiotemporal  Correlation  in  STMDs:  This 
goal  was  to  determine  experimentally  if  STMDs  exploit  the  longer-range  spatiotemporal 
correlations  that  arise  due  to  the  constraint  that  target  motion  occurs  along  continuous 
paths,  and  to  elucidate  the  nature  of  the  computations  by  which  this  is  achieved. 

AIM  3.  Further  Characterization  of  STMD  Behavior:  The  goals  included  determination  of  the 
mechanism  of  the  immense  amplification  required  to  support  vigorous  responses  to 
sub-pixel  targets;  how  inhibitory  responses  within  the  receptive  field,  or  on  earlier  pathways 
to  the  STMDs,  suppress  responses  to  other  features;  and  the  range  of  ‘non-target’  stimuli 
(e.g.  features  of  the  background  image)  that  produce  responses  from  the  STMD.  To  some 
extent,  this  goal  was  usurped  by  new  findings  that  our  experiments  were  recruiting  a 
selective  attention  mechanism  (see  AIM  6  below) 

AIM  4.  Improved  ‘Standard’  Front  End  for  Modeling:  The  goal  was  to  determine  to  what  degree, 
and  how,  photoreceptor  properties  increase  target  detectability,  to  enable  improved 
modeling  of  target  detection. 

AIM  5.  Improved  Model  for  Wide-Field  STMDs:  The  goal  was  to  develop  a  more  capable 
wide-field  STMD  model  that  incorporates  the  spatiotemporal  filtering  characteristics  of  early 
vision,  followed  by  rectification  and  adaptation  inherent  in  the  RTC,  and  a  final  stage  based 
on  a  temporal  nonlinear  facilitation  mechanism.  One  benchmark  for  this  model  will  be  to 
explain  target  motion  detection  without  relative  motion  cues. 

New  Aims  (as  proposed  in  July  2012): 

2  new  aims  (proposed  in  our  July  2012  report)  built  on  important  new  experimental  physiological 
results  obtained  during  the  earlier  part  of  this  grant: 

AIM  6.  Selective  Attention  in  STMD  neurons:  A  key  approach  used  of  the  earlier  reporting 
periods  for  this  project  to  address  AIM  3  were  experiments  that  involved  the  use  of  two 
targets,  at  variable  locations  within  the  receptive  field,  to  characterize  inhibitory  interactions 
operating  on  different  spatial  baselines.  These  inhibitory  interactions  tune  selectivity  for 


4 


small  features  and  assist  rejection  of  background  features.  We  published  results  addressing 
this  aim  during  the  first  3  performance  periods,  but  several  features  of  these  data  sets 
remained  difficult  to  explain  in  large  field  STMDs  such  as  dragonfly  CSTMD1  neurons.  In 
our  more  recent  experiments  we  tested  the  hypothesis  that  the  unexplained  variation  in  our 
data  results  from  the  2-target  experiment  paradigm  itself  recruiting  a  selective  attention 
mechanism.  The  results  obtained  during  the  last  part  of  this  grant  strongly  support  such  a 
mechanism. 

AIM  7.  Nonlinear  spatiotemporal  correlation  between  OFF  and  ON  edges:  In  our  paper 
describing  an  initial  model  for  the  STMD  (Weiderman  et  al  2008),  we  accepted  the 
emerging  view  that  early  visual  processing  in  insects  segregates  responses  into  ON  and 
OFF  channels,  and  postulated  a  mechanism  for  small  target  sensitivity  that  involves  a 
supra-linear  interaction  between  OFF  channel  response  (as  induced  by  leading  edge  of  a 
dark  target)  and  ON  channel  response  (as  induced  by  the  trailing  edge).  (In  particular,  this 
mechanism  accounts  for  sensitivity  to  smallness  of  the  target  in  the  direction  of  travel.) 
Direct  evidence  for  this  interaction  was  lacking,  however,  so  obtaining  such  evidence  and 
incorporating  it  into  revised  ESTMD  model  models  formed  the  basis  for  the  final  aim 
addressed  during  the  last  12  months  of  this  project. 

Status  of  the  major  project  aims 

Aims  1,  4  and  5  of  the  original  grants  were  largely  addressed  by  work  reported  in  early  progress 
reports  and  documented  in  our  final  report  for  FA9550-09-1-0116  (10  March  2012).  In  the  final 
performance  periods  for  FA2386-10-1-41 14  (which  ended  up  extended  well  beyond  the  original 
proposed  schedule)  we  were  able  to  make  substantial  progress  towards  addressing  both  aims  2 
and  3,  but  our  main  effort  was  devoted  to  the  new  aims  (6  &  7): 

•  Aim  6.  At  the  conclusion  of  this  project,  we  had  also  made  substantial  progress  towards 
addressing  aim  6.  Initial  results  were  described  in  our  interim  report  (July  2012).  A  first 
paper  was  subsequently  accepted  for  publication  in  the  prestigious  journal  Current  Biology 
(Wiederman  &  O'Carroll  2013a,  published  online  December  2012).  Our  most  recent  work 
addressing  this  new  aim  offers  the  potential  for  a  major  future  expansion  of  this  project  to 
use  insect  STMD  neurons  as  an  important  new  physiological  model  for  understanding 
mechanisms  of  competitive  selection  and  attention  in  response  to  multiple  distractor 
features.  Such  a  follow-on  effort  is  supported  from  early  2013  by  the  award  of  a  new  3-year 
project  grant  from  the  Australian  Research  Council. 

•  Aim  7:  We  have  now  tested  this  hypothesis  in  experiments  where  a  target  (with  both 
leading  and  trailing  edges)  is  drifted  across  the  receptive  field  of  the  neuron  and  responses 
compared  with  those  to  a  leading  ‘OFF’  edge  (an  ‘extending’  dark  bar  entering  the  display), 
or  a  trailing  ‘ON’  edge  (a  dark  bar  exiting  the  display)  alone.  We  have  further  addressed  this 
aim  by  elaboration  of  computational  models  that  capture  a  number  of  key  elements  of  the 
behavior  of  biological  STMDs,  including  those  arising  from  our  earlier  work  on  Aims  1  -5. 

3.  Experiment: 

The  experiments  described  below  focus  primarily  on  the  new  work  addressing  aim  7,  building 
substantially  on  the  pilot  data  included  in  the  interim  report  for  FA2386-10-1-41 14  (July  2012)  and 
our  final  report  for  FA9550-09-1-01 16  (10  March  2012).  This  experiment  work  was  largely  carried 
out  by  A/Prof  O’Carroll  and  supported  research  fellow  Dr  Steven  Wiederman  using 
electrophysiological  recording  facilities  methods  and  associated  computational  modeling  in  the 
Adelaide  laboratory.  This  work  was  recently  re-submitted  to  the  prestigious  Journal  of  Neuroscience 
(figures  1-7).  The  additional  modeling  documented  here  was  (Figures  8-12)  was  recently  submitted 
to  the  2013  IEEE  Symposium  on  Computational  Intelligence  for  Multimedia,  Signal  and  Vision 
Processing  (CIMSIVP). 

3.1  Nonlinear  spatiotemporal  correlation  between  OFF  and  ON  edges: 

Physiological  recording:  We  inserted  aluminium  silicate  glass  microelectrodes  (filled  with  2M  KOI, 
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80-120  MQ)  into  the  brain  of  immobilized  Hemicordulia  tau  (n=13,  either  sex).  We  recorded 
intracellularly  from  individual  neurons,  identifying  STMDs  from  their  size  tuning  and  characteristic 
receptive  field,  mapped  using  37  horizontal  and  21  vertical  scans  of  a  1 .25°  dark  target  across  a  120 
Hz  HD  LCD  monitor  (100°x80°  viewing  extent)  at  457s.  For  full  details  of  these  methods,  see 
Bolzon  et  al .,  2009.  To  study  selectivity  for  dark  targets,  we  drifted  1.25°x1.25°  targets  of  varying 
luminance  relative  to  the  background  at  457s  horizontally  through  the  strongest  region  of  the 
receptive  field.  Nominal  Weber  contrasts  were  calculated  from  RGB  values  (linearized  monitor, 
white  background  315  Cdm"2),  C  =  (Target  -  Background)  /  Background. 

ESTMD  model:  We  implemented  a  model  for  an  ESTMD  in  MATLAB  as  described  in  detail  in 
Wiederman  et  al.  (2008;  2009;  2010).  This  model  accounts  for  the  size  and  velocity  tuning  observed 
in  insect  STMDs  and  is  robust  in  the  presence  of  background  clutter,  even  without  relative  motion 
cues  (Wiederman  et  al.,  2008),  as  observed  in  physiological  responses  of  STMDs  (Nordstrom  et  al., 
2006,  Wiederman  et  al.,  2011).  Parameters  are  based  on  fly  physiology,  with  modification  to  match 
velocity  tuning  of  the  dragonfly  (Geurten  et  al.,  2007).  Briefly,  2-dimensional  spatial  input  is  optically 
blurred  (FWHM  1.4°)  and  hexagonally  sampled  (1°  inter-receptor  angle)  at  each  time  step  (IKHz). 
Photoreceptor  dynamics  are  based  on  the  model  proposed  by  van  Hateren  and  Snippe  (2001). 
Spatial  antagonism  of  the  1st  order  interneurons  (LMCs)  was  implemented  via  weighted  subtraction 
of  nearest  neighbours  in  the  hexagonally  sampled  inputs  to  allow  transmission  of  70%  static 
luminance  and  thus  match  the  weak  lateral  inhibition  described  from  LMC  recordings  in  the  same 
dragonfly  species  (Laughlin,  1974).  LMC  temporal  filters  were  derived  from  Juusola  et  al.  (1995) 
implemented  with  a  relaxed  high-pass  filter  (lower  corner  frequency  8  Hz)  and  a  small  DC 
component  (10%). 

The  LMC  output  was  fed  into  an  additional  stage  which  took  direct  inspiration  from  our 
electrophysiological  recordings  from  rectifying  transient  cells  (RTCs,  see  AIM  1)  described  from  the 
locust  medulla  (Osorio,  1991,  O'Carroll  et  al.,  1992),  and  1st  optic  chiasm  and  the  medulla  of  the 
blowfly  (Jansonius  and  van  Hateren,  1991,  Wiederman  et  al.,  2008).  In  these  RTCs,  transient  ON 
and  OFF  phases  (from  the  LMC  high-pass  filtering)  are  separated  via  a  further  temporal  high  pass 
filter  (t  =  100  ms  or  200  ms)  to  remove  any  sustained  signal  component  before  half-wave 
rectification  with  each  channel  exhibiting  independent  and  fast  adaptation.  Adaptation  states  for  this 
stage  are  determined  by  a  nonlinear  filter  which  approximates  cellular  ‘fast  depolarization  and  slow 
repolarization’  responses,  which  switches  its  time  constant  dependent  on  whether  the  input  is 
increasing  or  decreasing  (‘fast’,  t  =  3  ms,  when  channel  input  is  increasing  and  ‘slow’,  t  =  70  ms, 
when  decreasing).  A  key  property  that  results  from  inclusion  of  this  complex,  nonlinear  filtering  is  the 
selectivity  for  ‘novel’  transient  contrast  changes,  with  the  suppression  of  fluctuating  textural 
variations.  This  temporal  processing  thus  explains  the  robustness  of  STMD  neurons  to  contrasting 
targets  against  complex  backgrounds. 

The  next  model  stage  includes  strong  surround  antagonism  with  ON  channels  inhibiting  ON  and 
OFF  inhibiting  OFF,  as  clearly  observed  in  blowfly  RTCs  (Jansonius  and  van  Hateren,  1993).  This 
‘like’  channel  inhibition  is  essential  to  suppress  responses  for  features  that  are  extended  along  the 
axis  orthogonal  to  their  motion  (i.e.  the  ‘hypercomplex  property’  -  selectivity  for  small  objects  versus 
extended  bar  features,  Nordstrom  &  O’Carroll  2009).  The  RTC  output  then  combines  ON  and  OFF 
channels  via  a  facilitatory  interaction  between  the  delayed  OFF  channel  (low  pass  filter,  t  =  25  ms) 
and  the  undelayed  ON  channel.  This  is  modeled  with  a  multiplication  operation;  for  generality,  linear 
terms  may  also  be  included:  a»[ON]  +  b*[OFFdeiayed]  +  c*[ON]*[OFFdeiayed]- 

Comparison  of  ESTMD  and  EMD  models:  For  comparison  with  the  ESTMD,  EMD  models  were 
implemented  using  the  basic  schemes  described  for  a  2-detector  EMD  by  Eichner  et  al.  (201 1 )  and 
a  6-detector  EMD  by  Clark  et  al.  (201 1 )  except  that  these  were  implemented  using  the  same  optical 
blur,  hexagonal  sampling,  inter-receptor  angle,  and  early  visual  filtering  as  the  ESTMD  described 
above  (i.e.  photoreceptor  and  LMC  stages).  The  signal  was  then  half-wave  rectified  into  ON  and 
OFF  channels.  The  2-detector  EMD  model  sums  the  output  of  individual  correlations  between  ON 
with  ON  (‘LI  pathway',  Fig.  IB)  and  OFF  with  OFF  (‘L2  pathway')  channels.  The  6-detector  model 
(Fig.  1C)  includes  additional  correlations  between  opposite  signs  of  contrast  polarity  (weighted  as  in 
Clark  et  al.  201 1 ),  i.e.  correlations  between  ON  with  OFF  channels  as  well  as  between  OFF  with  ON 
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channels.  Both  EMD  and  ESTMD  models  were  implemented  as  a  hexagonal  grid  of  local  detectors 
spanning  an  80°x80°  field.  Data  in  Figure  6  were  from  a  single  ESTMD  within  this  grid,  centred  on 
the  target  trajectory.  Data  in  Figure  7  were  summed  over  the  entire  array  to  mimic  the  output  of  a 
higher-order  lobula  plate  tangential  neuron  and  allow  us  to  measure  responses  where  the  leading 
and  trailing  edges  of  the  bar  stimuli  may  be  widely  separated. 

4.  Results  and  Discussion: 

4.1  Separation  of  OFF  and  ON  pathways  in  insect  vision 

The  dominant  computational  model  for  biological  motion  processing  for  50  years,  the 
Flassenstein-Reichardt  (FIR)  model  for  an  elementary  motion  detector  (EMD),  involves  correlation  of 
spatially  separated  contrast  signals  after  delaying  one  channel  (Fig.  1  A).  This  model  is  supported  by 
diverse  evidence  from  behavioural  and  electrophysiological  studies,  particularly  in  dipteran  flies 
(Flassenstein  and  Reichardt,  1 956,  review  Borst  and  Euler,  2011),  but  its  specific  neural 
implementation  remains  elusive.  In  particular,  the  degree  to  which  contrast  signals  are  separated 
into  parallel  ON  and  OFF  pathways  (for  luminance  increments  and  decrements)  and  then 
recombined  within  the  EMD  has  been  questioned  for  many  years  (Egelhaaf  and  Borst,  1985; 
Franceschini  et  al. ,  1989).  Data  from  downstream  Vertical  and  Florizontal  System  neurons  used  for 
optic  flow  analysis  by  dipteran  flies  supports  a  2-detector  EMD  (Fig.  IB)  in  which  ON-ON  and 
OFF-OFF  information  is  correlated  locally  in  parallel  pathways  which  are  then  summed  (Eichner  et 
al.,  201 1 ,  Joesch  et  al.  2013). 


A  HR-EMD  B  2  Detector  EMD  C  6  Detector  EMD  D  ESTMD 


Si  S2 


Figure  1.  Correlation  based  models  of  elementary  motion  detection.  A,  The 
Flassenstein-Reichardt  EMD  correlates  spatially  separated  luminance  signals  (S-i  and  S2) 
with  one  path  delayed  in  time.  The  subtraction  of  a  mirror  symmetric  unit  provides  opponent 
direction  selectivity  to  the  EMD.  B,  In  the  2-detector  EMD  (Eichner  et  al.,  201 1 ),  like  channels 
are  correlated  with  one  another,  i.e.  ON  with  ON  (LI)  and  OFF  with  OFF  (L2).  LI  and  L2  refer 
to  types  of  laminar  monopolar  cell.  C,  The  6-detector  EMD  (Clark  et  al.,  2011)  has  more 
complex  combinatorial  interactions  between  channels,  including  correlation  between  spatially 
separated  OFF  and  ON  signals.  D,  In  the  ESTMD  model,  center-surround  antagonism  of  ON 
and  OFF  channels  is  followed  by  the  correlation  of  the  delayed  OFF  with  the  undelayed  ON 
signal. 

Insect  small  target  motion  detector  (STMD)  neurons  have  been  characterized  in  the  lobula  and 
lateral  midbrain  of  both  dragonflies  (O'Carroll,  1993,  Geurten  et  al.,  2007)  and  dipteran  flies 
(Nordstrom  et  al.,  2006;  Nordstrom  and  O'Carroll,  2006;  2009,  Barnett  et  al.,  2007).  Typical  STMDs 
respond  robustly  to  small  moving  targets  (subtending  ~1-3°)  even  against  background  clutter 
(Nordstrom  et  al.,  2006;  Wiederman  and  O'Carroll,  201 1 ).  In  an  attempt  to  explain  these  properties, 
we  previously  proposed  a  model  for  an  elementary  small  target  motion  detector  (ESTMD) 
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employing  a  correlation  of  ON  signals  at  each  location  with  delayed  OFF  to  match  the  expected 
signature  of  a  small,  dark  feature  (Fig.  ID): 

Despite  the  ESTMD  model  providing  an  excellent  fit  to  the  spatiotemporal  tuning  of  STMD  neurons, 
several  key  predictions  of  it  remain  untested.  Our  experiments  in  the  final  phase  of  this  grant  aimed 
to  address  this  deficiency  via  recordings  both  from  CSTMD1  and  from  several  other  types  of 
dragonfly  STMD.  We  obtained  evidence  for  a  potent  non-linear  interaction  between  OFF  and  ON 
channels  in  this  alternative  motion  pathway.  We  show  that  many  STMD  neurons  only  respond  to 
dark  objects,  with  little  or  no  response  to  light  objects  with  equal  contrast.  These  responses  are 
greater  than  predicted  from  the  linear  combination  of  responses  to  dark  or  light  edges  of  identical, 
limited  lateral  extent.  Finally,  we  show  that  classical  HR-EMD  models  (either  with  or  without  strong 
surround  antagonism)  cannot  account  for  our  data,  but  our  model  of  STMD  neurons  is  well  matched 
(Wiederman  et  al.,  2008;  2010).  Thus  we  provide  evidence  that  feature-specific  information  is 
extracted  by  operations  involving  the  supralinear  combination  of  ON  and  OFF  contrast  pathways  in 
the  dragonfly  brain. 

ESTMD  model 

Although  variants  of  the  HR  model  (Fig.  1A)  have  previously  been  used  to  model  spatiotemporal 
tuning  of  STMD  neurons  (Geurten  et  al.,  2007,  Dunbier  et  al.,  201 1;  2012),  these  models  lack  the 
size  selectivity  that  characterizes  STMDs.  We  therefore  previously  proposed  the  ESTMD  (Fig.  ID) 
as  a  model  for  a  fundamental,  local  signal  processing  stage  underlying  STMD  behavior  (Wiederman 
et  al.,  2008;  2009),  which  is  then  summed  spatially  by  wider-field  STMD  neurons  amenable  to 
electrophysiological  study. 

Like  the  2-detector  EMD  proposed  by  Eichner  et  al.  (201 1 )  (Fig.  1 B),  our  ESTMD  model 
incorporates  separation  of  half-wave  rectified  ON  and  OFF  channels.  In  Drosophila,  behavioral 
responses  provide  evidence  for  additional  OFF-ON  and  ON-OFF  interactions  (Fig.  1 C)  that 
characterize  a  more  complex  6-detector  EMD  model  (Clark  et  al.,  201 1 ).  Unlike  the  2-detector  EMD, 
but  as  in  two  subunits  of  the  6-detector  model,  the  ESTMD  involves  correlation  of  opposite  sign 
channels  (Fig.  ID).  Unlike  any  variant  of  the  EMD  proposed  to  date,  however,  the  ESTMD 
correlates  ON  with  delayed  OFF  signals  arising  at  the  same  retinotopic  location,  rather  than  from 
adjacent  or  nearby  detector  pairs.  This  exploits  properties  of  a  spatially  circumscribed  feature 
moving  in  a  given  direction:  even  against  cluttered  backgrounds  such  a  feature  is  likely  to  have  a 
leading  edge  luminance  change  opposite  in  sign  to  its  trailing  edge.  A  tiny  dark  object  crossing  the 
receptive  field  of  a  single  photoreceptor  would  thus  produce  a  response  that  first  falls  before  rising. 

The  addition  of  strong  surround  antagonism  within  the  ESTMD  further  enhances  small  target 
selectivity  along  the  axis  orthogonal  to  its  motion,  while  rapid  adaptation  in  the  rectifying  ON/OFF 
elements  rejects  repetitive  local  flicker  stimuli  induced  by  background  texture.  Although  STMD 
neurons  show  varying  degrees  of  direction  selectivity  (O’Carroll  1993,  Nordstrom  et  al.  2006, 
Barnett  et  al.  2007)  the  ESTMD  model  as  originally  formulated  (Wiederman  et  al.,  2008)  is 
inherently  non-directional.  The  inclusion  of  direction  selectivity  in  the  ESTMD,  however,  can  be 
readily  modeled  by  several  alternative  mechanisms  without  changing  the  fundamental  selectivity  for 
small  features  (Wiederman  &  O’Carroll  2013a).  These  include  spatial  separation  of  the  correlated 
ON  and  OFF  channels,  asymmetry  in  the  inhibitory  surround,  or  2nd  order  ESTMD-like  correlation  of 
inputs  derived  from  2-detector  EMDs  (or  vice  versa). 

Selectivity  for  dark  features 

An  untested  prediction  arising  from  the  opposite-polarity  correlation  in  the  ESTMD  model  is  its 
selectivity  for  the  sign  of  the  contrast  ‘signature’  produced  by  a  given  target.  The  detection  of  both 
light  and  dark  targets  would  require  local  ESTMD  pairs  to  correlate  both  delayed  ON  with  OFF,  and 
delayed  OFF  with  ON  channels,  while  a  simpler  ESTMD  (as  in  Figure  ID)  would  be  selective  for 
dark  features.  To  test  whether  this  is  the  case,  we  quantified  contrast  sensitivity  of  1 0 
feature-detecting  neurons  by  drifting  small  light  (i.e.  an  ON-OFF  stimulus)  or  dark  (OFF-ON)  targets 
through  the  receptive  field.  We  presented  different  combinations  of  stimuli  comprising  ‘OFF’  and 
‘ON’  edges,  as  well  as  ‘Target’  features.  Figure  2  describes  these  stimuli  in  detail.  In  each  panel,  the 
upper  pictograms  describe  the  stimuli  at  a  single  point  in  time  (i.e.  a  diagrammatic  ‘snapshot’  of  our 
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stimulus  screen).  The  lower  panel  shows  a  space/time  plot  of  the  corresponding  stimulus,  i.e.  the 
luminance  change  along  the  line  that  the  feature  moves  along.  Each  stimulus  had  identical,  limited 
spatial  extent  (1 .25°)  in  the  axis  orthogonal  to  its  motion.  These  were  each  presented  at  4  different 
locations  within  the  -80°  degree  wide  receptive  field  of  CSTMD1  (at  5°  separations).  We  included  a 
minimum  rest  interval  of  50  seconds  between  stimuli  presented  at  the  same  receptive  field  location. 
This  was  aimed  at  minimizing  local  habituation  effects.  Edges  and  targets  were  presented  at  high 
contrast  (Fig.  2A),  low  contrast  (Fig.  2B)  and  at  both  contrast  polarities  (Fig.  2C). 

In  6  recordings  from  the  identified  neuron,  CSTMD1  (Fig.  3A)  and  4  from  unidentified  STMDs  (Fig 
3B)  we  observed  profound  dark  target  selectivity.  Figure  3A  shows  mean  spike  rate  for  an  example 
CSTMD1  neuron  (within  a  500ms  analysis  window  centered  on  the  receptive  field)  in  response  to  a 
range  of  target  contrasts  of  both  light  and  dark  polarities  (mean  ±  SEM  of  5  trials).  As  targets  get 
darker,  spike  rate  increases.  Flowever  responses  to  light  targets  elicit  minimal  response  even  at  the 
highest  contrast.  Data  averaged  across  5  CSTMD1  neurons  (Fig.  3A,  inset)  confirmed  selectivity  for 
the  polarity  of  the  stimulus,  with  a  significantly  larger  response  to  dark  (CWeber  =  -1 )  versus  light 
(Cweber =  1)  targets  (P=0.009,  paired  t-test).  Indeed,  responses  to  light  targets  are  not  significantly 
above  spontaneous  levels. 
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Figure  2.  Visual  stimuli  presented  to  the  dragonfly  during  electrophysiological  recordings 
from  STMD  neurons.  Upper  rows  of  each  combination  depict  a  snapshot  in  time  of  the  2-d 
display  on  which  stimuli  were  presented  (not  to  scale)  while  lower  rows  are  space-time  plots 
for  the  luminance  change  along  the  line  that  the  feature  moves.  A,  High  contrast  OFF  and  ON 
edges  and  a  single  black  target  are  drifted  from  the  bottom  to  the  top  of  the  display.  B,  Low 
contrast  versions  of  both  edges  and  target,  with  the  addition  of  a  ‘target’  composed  of  two 
OFF  edges.  C,  On  a  mean  background,  ON  and  OFF  edges  are  composed  from  both  mean 
background  to  light  and  mean  background  to  dark  transitions.  Both  light  and  dark  targets  are 
also  presented. 

The  four  unidentified  feature-selective  neurons  (Fig.  3B)  all  show  varying  degrees  of  size  selectivity 
for  targets  or  short  bars  (determined  from  size-tuning,  Fig.  3B,  inset)  between  1  and  10°  (i.e. 
orthogonal  to  the  direction  of  travel).  In  all  of  these  feature-selective  neurons,  darker  targets  evoked 
robust  responses  saturating  at  relatively  low  contrasts  (CWeber  between  -0.2  and  -0.5).  They  all  elict 
very  weak  responses  to  high  contrast  light  targets  (Fig.  3B),  thus  confirming  that  dark  target 
selectivity  is  not  unique  to  CSTMD1 .  If  we  present  light  targets  against  a  black  screen  background 
(a  stimulus  that  represents  a  much  higher  Weber  contrast)  we  do  see  intermittent  weak  responses 
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(data  not  shown).  We  excluded  this  data  from  our  quantitative  analysis,  however,  because  it 
subjects  the  photoreceptors  to  a  significantly  lower  adapting  luminance,  well  below  the 
physiologically  normal  range. 
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Figure  3.  Dark  target  selectivity  in  the  dragonfly  brain.  A,  An  individual  example  of  CSTMD1 
responses  to  targets  of  varying  contrast  moving  on  a  mid-luminance  background  (mean±SEM, 

5  trials).  Responses  strengthen  as  negative  contrast  polarity  of  the  target  increases  (dark), 
however  positive  contrast  (light)  targets  elicit  little  response.  Across  5  neurons  (inset  graph, 
mean  ±  SEM),  CSTMD1  displays  dark  target  selectivity,  responding  significantly  more 
(P=0.009)  to  high  contrast  black  targets  (CWeber  =  -1)  than  to  high  contrast  white  targets 
(CWeber  =  1),  which  are  indistinguishable  from  spontaneous  activity.  B,  Four  unidentified 
neurons  in  Hemicordulia  tau  produce  robust  responses  to  targets  of  increasing  dark  contrast 
but  not  to  light  contrast  (left  ordinate:  black  squares  and  grey  diamonds;  right  ordinate:  grey 
triangles  and  white  circles).  These  unidentified  feature-selective  neurons  exhibit  varying 
size-selectivity  (inset  graph). 

Supralinear  summation  of  edge  responses 

From  Figure  3,  we  conclude  that  STMD  neurons  show  dark  target  selectivity  consistent  with 
predictions  of  an  ESTMD  model  that  correlates  delayed  OFF  with  ON  information.  To  further  test 
whether  this  involves  a  supralinear  interaction  between  these  channels,  we  presented  STMD 
neurons  with  moving  ON  and  OFF  features  with  limited  extent  in  the  dimension  orthogonal  to  their 
motion  (i.e.  local  edge  features)  and  compared  responses  with  those  for  the  equivalent  discrete 
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targets.  These  stimuli  are  illustrated  by  the  pictograms  in  Figures  2  and  4.  Figure  4A  shows  mean 
peristimulus  time  histograms  from  two  individual  CSTMDIs  in  response  to  the  different  stimuli.  In 
the  first  example,  we  repeated  each  stimulus  5  times  at  4  locations  within  the  receptive  field  (20 
trials).  Both  OFF  edge  (dashed  line)  and  ON  edge  (dotted  line)  responses  are  weak.  Target 
responses  (solid  line)  are  greater  than  the  linear  combination  of  the  ON  and  OFF  edge  responses. 
In  a  second  CSTMD1  example  (16  trials),  OFF  edges  produce  moderate  responses  that  are 
stronger  than  ON  edge  responses.  Figure  4B  shows  the  responses  to  the  same  stimuli  for  one  of 
the  unidentified  neurons  (STMD-U1),  which  also  resulted  in  a  supralinear  target  response. 


A  High  Contrast  B  C 


Figure  4.  A ,  Two  examples  of  individual  CSTMD1  responses  to  high  contrast  stimuli  drifted 
through  the  receptive  field.  In  the  first,  CSTMD1  exhibits  little  response  to  either  ON  or  OFF 
edges  and  the  response  to  targets  is  larger  than  a  linear  combination  of  the  edge  responses 
(mean  of  20  trials).  In  the  second  example,  CSTMD1  responds  to  high  contrast  ON  and  OFF 
edges,  though  both  are  weaker  than  target  responses  (mean  of  16  trials).  B,  The  response  of 
an  un-identified  STMD  neuron  to  the  same  stimuli  exhibits  a  similar  supralinearity  (mean  of 
20  trials).  C,  Pooled  CSTMD1  responses  (68  trials  over  5  neurons)  show  median  target 
responses  are  larger  than  the  linear  combination  of  responses  to  ON  and  OFF  edges  (Target 
vs.  ON  +  OFF,  P  =  0.001 ,  Mann-Whitney  U).  In  some  trials,  edge  responses  evoke  responses 
from  CSTMD1  in  particularly  to  the  OFF  stimulus.  D,  CSTMD1  responses  to  lower  contrast 
target  and  edges  stimuli  (CWeber  =  -0.6)  and  a  ‘double’  OFF  edge  stimulus  (white  to  grey  to 
black).  An  example  CSTMD1  response  to  the  lower  contrast  stimuli  exhibits  strong  target 
responses  and  weak  response  to  either  ON  or  OFF  edges.  Furthermore,  the  double 
OFF/OFF  stimulus  elicits  minimal  response  (average  of  12).  E,  CSTMD1  target  responses 
are  greater  than  any  linear  combination  of  the  ON  and  OFF  edge  responses  (16  trials  over  2 
neurons). 

To  examine  reproducibility  of  responses  in  CSTMD1 ,  we  pooled  results  from  5  neurons  (Fig.  4C,  Is 
analysis  window,  68  trials).  There  is  a  significant  difference  between  all  4  conditions:  P<0.05  (ON  vs. 
OFF),  P<0.001  (others),  Dunn’s  multiple  comparison,  Kruskal-Wallis.  Furthermore,  target 
responses  are  significantly  higher  than  the  sum  of  the  ON  and  OFF  edge  responses  (P  =  0.001 , 
Mann  Whitney  U),  thus  supporting  a  supralinear  interaction  between  the  OFF  and  ON  channels. 

With  the  high  contrast  stimuli  in  Figure  4A-C,  many  individual  trials  produce  strong  edge  responses, 
particularly  to  the  OFF  stimulus,  and  target  responses  may  easily  exceed  300  spike/s.  We  briefly 
discussed  this  variation  in  our  previous  progress  report  (July  2012)  for  this  project  and  speculated 
as  to  possible  causes.  In  particular,  saturation  could  potentially  mask  the  full  degree  of  the 
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non-linear  interaction  between  OFF  and  ON  channels.  We  therefore  also  presented  lower  contrast 
versions  of  the  stimuli  (see  Fig.  2B)  to  2  CSTMD1  neurons  (16  trials  total).  This  allowed  us  to  also 
test  a  further  variant,  a  ‘double’  OFF-edge  stimulus  (also  mentioned  in  the  previous  report).  This 
comprised  a  white  to  grey  transition  on  the  leading  edge  of  the  feature,  followed  by  a  further  grey  to 
black  transition,  separated  by  1 .25°  (Fig.  2B).  This  feature  induces  luminance  changes  similar  in 
contrast  and  spatial  extent  to  the  low  contrast  dark  target,  except  that  the  polarity  of  the  trailing  edge 
is  the  same  as  the  leading  edge. 

Figure  4D  shows  mean  responses  to  these  stimuli  in  an  example  CSTMD1,  revealing  potent 
supralinearity  of  the  target  response  relative  to  ON  and  OFF  edge  responses.  When  pooled  over  the 
entire  set  of  1 6  trials  (Fig.  4E),  the  mean  target  responses  are  much  larger  than  for  any  of  the 
other  conditions,  including  the  double  OFF-edge  stimulus.  This  suggests  that  the  target  response 
requires  both  ON  and  OFF  channels. 

In  order  to  test  stimuli  for  both  light  and  dark  contrast  edge  polarities  we  also  presented  low  contrast 
stimuli  against  a  mean  luminance  (grey)  background  (Fig.  2C)  in  two  further  CSTMD1  neurons  (Fig. 
5A,  12  trials).  Dark  target  selectivity  is  again  evidenced  by  the  much  stronger  response  to  dark 
targets  than  any  other  combination  of  light  or  dark  contrasting  edges.  Some  of  the  variation  evident 
in  the  box-plot  distributions  in  Figure  5A  is  due  to  the  inhomogeneous  sensitivity  at  different 
locations  within  the  receptive  field  over  which  stimuli  were  presented.  Figure  5B  shows  individual 
responses  to  low  contrast  dark  targets  plotted  against  the  sum  of  the  OFF  and  ON  edge  responses 
matched  to  the  same  location  in  the  receptive  field.  Observed  responses  to  low  contrast  dark  targets 
are  clearly  always  much  stronger  (dashed  line,  slope  =  1.9±0.4  [95%CI])  than  the  predictions  of  the 
linear  sum  of  ON  and  OFF  responses  (solid  line,  slope=1 ). 
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Figure  5.  CSTMD1  responses  to  low  contrast  edge  and  target  stimuli  A,  Irrespective  of  the 
edge  transition  with  respect  to  the  background  (i.e.  ON,:  dark  to  mean  or  ON2:  mean  to  light; 
OFF^  mean  to  dark  or  OFF2:  light  to  mean),  CSTMD1  responses  in  a  further  2  neurons  (12 
trials  total)  are  much  stronger  to  the  dark  target.  These  dark  target  responses  are  much  larger 
than  those  to  the  light  target  or  the  linear  combination  of  either  of  the  ON  and  OFF  edge 
combinations.  B,  CSTMD1  responses  to  low  contrast  dark  targets  are  plotted  against  the 
linear  sum  of  ON  and  OFF  responses  (matched  at  the  same  receptive  field  location).  Solid 
line  represents  a  linear  combination  (slope  of  1),  dashed  line  of  best  fit  (slope  =  1.9±0.4 
[95%CI]). 

Evidence  for  an  ESTMD-like  mechanism 

Taken  together,  our  experimental  data  confirm  that: 

(1 )  Only  dark  targets  evoke  robust  spiking  activity  from  CSTMD1 , 

(2)  The  target  response  involves  a  potent  supralinear  interaction  between  nearby  edges. 

(3)  The  target  response  requires  both  OFF  and  ON  components. 

Can  we  reproduce  these  results  with  the  ESTMD  model  that  we  previously  proposed  (Wiederman  et 
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al.,  2008)?  Figure  6  shows  data  for  ESTMD  models  that  include  ON  and  OFF  channels  that  interact 
both  linearly  and  multiplicatively:  a»[ON]  +  ft* [OF Flayed]  +  c»[ON]*[OFFdeiayed].  The  data  show 
distributions  for  outputs  of  our  model  in  response  to  similar  stimuli  to  those  presented  to  STMD 
neurons  in  Figures  4  and  5  (i.e.  at  both  high  and  low  contrast)  and  using  combinations  of 
coefficients  a,b  and  c  varied  over  a  large  range:  a  {0,  1 , 2};  b{ 0,1, 2};  and  c  {1 , 2,  5}.  These 
combinations  of  term  coefficients  allow  us  to  represent  different  ‘balances'  of  interactions  between 
separate  ON  and  OFF  channels. 
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Figure  6.  An  ESTMD  model  includes  terms  for  separate  ON  and  OFF  channels,  as  well  as 
a  facilitatory  correlation  between  the  delayed  OFF  and  undelayed  ON  signals.  a»[ON]  + 
b»[OFFdeiayed]  +  c»[ON]»[OFFdeiayed].  We  varied  a,  b  and  c  coefficients  and  examine  their 
pooled  effect  on  each  distribution  of  ON  edge,  OFF  edge  and  target  responses.  All  model 
variants  predict  supralinear  target  responses,  A,  High  contrast  stimuli  and  more  sustaining 
high  pass  filtering  (t=200itis),  predicts  either  or  both  ON  and  OFF  edge  responses.  S,  High 
contrast  and  more  transient  high  pass  filtering  (T=100ms)  produces  larger  OFF  edge 
responses,  as  observed  in  experiments  (Fig.  6B  cf.  Fig.  4C).  C,  For  lower  contrast  stimuli  the 
ESTMD  model  predicts  minimal  ON  or  OFF  edge  responses,  again  matching  the 
physiological  data  (Fig.  6C,D  cf.  Fig.  4E,5A). 


As  can  be  seen  from  Figure  6,  the  ESTMD  model  captures  not  only  the  supra-linear  interaction 
between  the  ON  and  OFF  components  of  the  target  stimulus,  it  also  predicts  the  presence  of  weak 
responses  to  the  ON  and  particularly  the  OFF  edges  at  high  contrasts  (Fig.  6A,B),  as  we  observed 
in  the  STMD  recordings  (Fig.  4,5).  This  is,  perhaps,  surprising  given  that  the  distributions  include 
combinations  of  term  coefficients  where  a=b= 0,  i.e.  a  purely  multiplicative  ESTMD.  Flowever,  with 
appropriate  temporal  high  pass  filtering  in  early  vision,  a  single  edge  stimulus  can  potentially  induce 
excitation  of  both  ON  and  OFF  channels  due  to  response  rebound  following  the  initial  luminance 
change.  Even  in  the  photoreceptors  themselves,  luminance  step  responses  may  show  complex 
multiphasic  transients  at  high  contrasts  even  though  they  encode  luminance  steps  faithfully  at  low 
contrasts  (Laughlin  and  Weckstrom,  1993).  Such  complex  temporal  filtering  would  lead  to  weak 
outputs  from  even  a  purely  multiplicative  ESTMD.  In  addition  to  varying  the  term  coefficients,  we 
further  tested  2  model  variants  (t=200itis,  t=1  00ms)  that  simulate  variation  in  high  pass  filtering  that 
could  represent  differences  in  the  state  of  light  adaptation  (van  Flateren  and  Snippe,  2001;  Juusola 
et  al.,  1995)  or  even  a  change  in  temperature  (Tatler  et  al.,  2000).  At  low  contrasts  (Fig.  6  C,D)  the 
model  is  relatively  robust  against  the  differing  contributions  of  the  term  coefficients  within  the  range 
we  considered,  and  always  shows  the  strongly  supralinear  interaction  effect  as  seen  in  the 
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physiological  data  (Fig.  4,  5).  The  faster  time  constant  model  better  matches  the  physiological  data, 
however,  in  that  it  gives  stronger  responses  to  dark  edges  at  high  but  not  lower  contrast  (compare 
Fig.  6B,D  with  Fig.  4C,E). 

Comparison  with  EMD  models 

From  Figure  6  we  conclude  that  with  careful  attention  to  the  temporal  properties  of  high-pass  filters 
that  are  a  key  element  of  the  model,  an  ESTMD-like  mechanism  can  explain  selectivity  for  dark 
features  and  the  supralinear  interaction  between  nearby  ON  and  OFF  edges  that  we  observe  in 
STMD  neurons.  Given  that  it  also  includes  a  multiplicative  interaction,  could  a  simple 
Flassenstein-Reichardt  correlation  EMD  be  an  alternative  explanation  for  our  data? 
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Figure  7.  Edges  (ON  &  OFF)  and  dark  targets  serve  as  input  to  model  variants  of  the  EMD  as 
well  as  the  ESTMD  model.  These  include  both  the  2-detector  EMD,  which  sums  channels  of 
‘like’  correlations  (Eichner  et  al .,  2011)  and  the  6-detector  EMD,  which  includes  terms 
between  ON  and  OFF  correlations  (Clark  et  al.,  2011).  Each  of  these  versions  are  modeled 
with  and  without  strong  surround  antagonism.  For  each  model  variant,  output  is  normalized  to 
the  sum  of  the  OFF  edge,  ON  edge  and  target  output  values.  A,  Without  surround 
antagonism  both  the  2  and  6-detector  models  predict  a  sublinear  response  to  targets  rather 
than  the  supralinearity  observed  in  CSTMD1  (Fig.4C,  E  and  Fig.  5).  B,  Model  outputs  do  not 
change  when  surround  antagonism  is  included  in  the  model  variants.  C,  In  comparison,  an 
STMD  model  matches  the  physiological  data  with  target  responses  much  larger  than  the 
linear  combination  of  ON  and  OFF  edge  responses.  D,  EMD  and  STMD  model  output  in 
response  to  targets  of  varying  extent  (in  the  direction  of  travel).  The  ESTMD  model  is  more 
responsive  to  small  targets  and  the  output  of  the  OFF  and  ON  correlation  is  inherently  tuned 
to  the  separation  of  the  OFF  and  ON  edges  in  the  direction  of  travel  (i.e.  velocity/width  tuned). 

On  the  contrary,  corresponding  models  for  both  a  2  and  6-detector  EMD  yield  sublinear  addition  of 
the  single  edge  responses  compared  with  the  response  to  the  tiny  targets  used  in  our  experiments 
(Fig.  7A,B).  This  is  in  contrast  to  the  output  supralinearity  observed  in  our  ESTMD  model  (Fig.  7C). 
This  is  because  optical  blur  by  the  facet  lenses  (Horridge,  1978,  Nordstrom  et  al.,  2006)  has  a 
greater  effect  on  the  small  targets  compared  with  the  single-edge  stimuli,  which  are  blurred  only 
across  their  narrow  dimension.  Therefore,  in  comparison  to  the  moving  edges,  target  responses  are 
expected  to  be  weaker  at  the  earliest  stages  of  visual  processing  despite  the  inclusion  of  a  front-end 
for  all  3  model  variants  including  processing  that  accounts  well  for  both  linear  and  non-linear 
spatiotemporal  filtering  in  early  visual  processing  (Laughlin  1974,  van  Flateren  &  Snippe  2001, 
Brinkworth  et  al.  2008,  Mah  et  al.  2008,  Wiederman  et  al.  2010).  This  makes  the  observed  strength 
of  the  response  to  the  target  versus  single  edges  in  the  STMD  recordings  all  the  more  impressive. 

Arguably,  the  contrast  attenuation  of  small  features  could  be  partially  offset  by  the  strong  surround 
antagonism  that  represents  an  additional  stage  of  the  ESTMD  model.  We  tested  this  by  simulation 
of  additional  variants  of  the  2  and  6-detector  EMD  models  that  include  this  extra  stage  of  processing 
(Fig.  7B).  Again  target  responses  are  weaker  than  the  linear  combination  of  the  single  edge 
responses.  If  we  extend  the  target  width  in  the  direction  of  travel  (Fig.  7D),  optical  blur  has  less 
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effect  on  the  target,  which  begins  to  resemble  a  double  edge  stimulus  (i.e.  an  OFF  leading  edge 
widely  separated  from  an  ON  trailing  edge).  With  two  edge  features  within  the  scene,  both  EMD 
variants  tested  then  yield  responses  greater  than  ON  or  OFF  edge  stimuli  alone,  but  never  exceed 
the  linear  combination  of  ON  and  OFF  edge  responses  (Fig.  7D).  The  ESTMD,  by  comparison, 
responds  progressively  more  weakly  (as  observed  in  STMD  neurons)  as  the  leading  and  trailing 
edge  become  further  separated  in  time. 

4.2  Velocity  tuning  and  contrast  polarity 

STMDs  are  tuned  to  the  velocity  of  a  moving  target.  This  property  is  itself  indicative  of  an  underlying 
mechanism  that  likely  involves  some  form  of  spatio-temporal  correlation  (Geurten  et  al.  2009; 
Nordstrom  and  O’Carroll,  2006).  Although  we  previously  suggested  this  provides  evidence  for  an 
FIR-EMD  framework,  the  result  is  equally  consistent  with  the  ESTMD-based  correlation  model.  In 
any  such  model,  a  pronounced  velocity  optimum  is  determined  by  early  visual  filtering  and  the 
correlation  delay  time  constant  (Geurten  et  al.,  2009;  Dunbier  et  al.,  2012),  with  the  optimum 
velocity  increasing  as  target  width  increases  in  the  direction  of  travel  (Geurten  et  al.,  2009).  This  is 
due  to  the  increased  spatial  separation  between  the  leading  and  trailing  edge  of  the  target,  requiring 
a  faster  transit  speed  to  match  a  given  delay  between  OFF  and  ON  channels.  The  ESTMD,  as  a 
‘temporal’  correlator,  can  extract  this  weak  rising  and  falling  signature  of  a  target  even  though  it 
primarily  occurs  in  a  single  ommatidium  at  a  time.  Although  directionality  may  be  readily  explained 
with  the  inclusion  of  spatial  interactions  or  asymmetry  (Wiederman  and  O’Carroll,  2013a),  a  benefit 
of  (purely  local)  temporal  processing  is  the  robust  responses  to  slowly  moving  targets  smaller  than  a 
single  ommatidium  -  a  simple  form  of  hyperacuity.  In  fact,  many  STMD  neurons  (including  CSTMD1 ) 
are  responsive  to  target  sizes  of  less  than  0.5°  square,  which  are  of  extremely  low  effective  contrast 
(Eristalis  tenax,  Nordstrom  et  al.,  2006;  Hemicordulia  tau,  Wiederman  &  O'Carroll  2013b). 

Our  modeling  shows  that  the  pattern  of  responses  observed  in  STMD  neurons  to  single  edge  versus 
target  features  is  poorly  predicted  by  HR-EMDs,  but  well  predicted  by  an  ESTMD  mechanism.  A 
side  effect  of  the  very  high  gain  required  to  achieve  the  observed  sensitivity  for  the  blurred  image  of 
very  small  targets  is  some  sensitivity  to  high  contrast  edges  with  limited  angular  extent.  Natural 
scenes,  however,  are  dominated  by  larger  features  (Dror  et  al.,  2000)  and  our  own  recording  from 
STMDs  in  response  to  natural  scenes  shows  that  breakthrough  responses  from  textural  features  of 
the  background  are  weak  even  in  strong  clutter  (Wiederman  &  O’Carroll,  201 1 ).  To  account  for 
some  ON  and  OFF  edge  excitation,  we  include  linear  terms  in  our  phenomenological  model.  What 
would  the  three  terms  of  our  model  represent  physiologically?  They  suggest  a  mechanism  of 
interaction  that  involves  weak  excitation  by  the  individual  signals  that  impinge  on  the  individual  arms 
of  the  correlator,  with  significant  enhancement  of  excitation  by  their  co-activation.  Significant 
variation  in  single-edge  sensitivity  seen  in  CSTMD1  from  animal  to  animal  could  reflect  varying 
excitation  thresholds  or  variation  in  the  strengths  of  the  individual  inputs  relative  to  their  degree  of 
mutual  facilitation. 

Flawking  dragonflies,  such  as  Hemicordulia  tau,  remain  continuously  in  flight,  swooping  upwards  to 
catch  their  prey  overhead  (Corbet,  1999).  In  this  scenario,  the  target  prey  is  located  within  an  area 
of  high  visual  acuity  (Florrige,  1978)  and  will  appear  dark  against  a  light  sky  background.  Flowever, 
we  also  observe  dragonflies  chasing  prey  and  conspecifics  against  cluttered  surrounds,  with  the 
changing  background  causing  the  perceived  target  to  alternate  between  light  and  dark  contrasts. 
From  the  behavioral  perspective,  it  would  seem  that  neurons  responsive  to  either  or  both  light  and 
dark  targets  should  exist.  Our  encountering  of  only  dark  target  selective  neurons  to  date  may  be 
coincidental  or  possibly  the  result  of  bias  in  our  experimental  preparation,  which  is  geared  towards 
recording  neurons  with  dorsally-centered  receptive  fields.  From  the  modeling  perspective,  should 
light-target  selectivity  be  observed  in  future,  this  problem  is  readily  solved  with  the  inclusion  of  an 
[ON delayed] *[  OFF]  term  to  form  a  light-target-sensitive  ESTMD. 

Although  we  present  results  from  several  types  of  STMD  neurons,  our  primary  evidence  is  derived 
from  recordings  of  CSTMD1.  This  neuron  has  visual  inputs  in  the  mid-brain  and  has  recently  been 
shown  to  exhibit  several  ‘higher-order’  attributes.  These  include  the  ability  to  selectively  attend  to  a 
single  target  in  the  presence  of  a  distractor  (Wiederman  and  O’Carroll,  201 3b),  as  well  as  exhibiting 


a  slow  facilitation  of  its  response  for  targets  that  move  along  a  continuous  trajectory  (Nordstrom  at  al. 
201 1 ;  Dunbier  et  al.  2012).  However,  CSTMD1  is  a  neuron  that  responds  very  quickly  in  absolute 
terms,  with  a  response  latency  less  than  50  ms  (Nordstrom  et  al.  2011)  and  tuning  to  high  target 
velocity  (Geurten  et  al.  2007;  Dunbier  et  al.  2012).  Our  modeling  of  its  response  kinetics  suggests 
that  the  facilitation  must  be  due  to  a  higher  order  property  within  the  STMD  pathway  (e.g.  attentional 
modulation)  in  response  to  slower  moving  features,  rather  than  any  inherent  sluggishness  in  the 
time  constants  of  the  underlying  motion  pathway  (Dunbier  et  al.  2011;  2012).  Thus,  some  caution  is 
still  in  order  in  attributing  the  observed  supralinear  response  characteristics  of  CSTMD1  to  an 
underlying  elementary  correlation  between  local  OFF  and  ON  channels,  versus  a  mechanism  that 
up-regulates  attention  for  the  highly  salient  target  stimulus.  Nevertheless,  the  fact  that  we  also 
observed  similar  supralinearity  in  several  other  unidentified  STMD  neurons,  suggests  that  this  is  a 
common  property  of  the  input  pathway. 


4.3  SF-STMDs,  Direction  selectivity  higher  order  facilitation:  could  a  2nd  order  detector 
provide  a  unified  solution? 

While  our  data  above  provides  strong  evidence  for  an  ESTMD-like  mechanism  being  involved  in  the 
STMD  neuronal  pathway,  it  falls  short  of  explaining  all  observed  properties  of  typical  STMDs.  In 
particular,  while  CSTMD1  itself  is  not  direction-selective,  other  STMDs  are.  Indeed,  successful 
tracking  and  interception  of  targets  must  involve  additional  information  about  both  direction  and 
location  of  the  target.  Secondly,  the  non-linear  facilitation  mechanism  that  we  have  described  in 
detail  in  earlier  reporting  periods  (and  the  associated  papers)  is  not  an  emergent  property  of  local 
ESTMD  processing.  It  must  result  instead  from  interactions  occurring  over  larger  spatial  baselines  - 
i.e.  between  ESTMDs  (or  their  output  collators)  themselves.  An  interesting  hypothesis  to  explain 
both  phenomena  is  that  they  employ  a  second-order  motion  detector  that  cascades  an  HR-type 
EMD  stage  with  an  ESTMD  stage.  This  is  further  suggested  by  the  our  discovery  of 
direction-selective  STMD  neurons  with  relatively  small  receptive  fields,  around  5-10°  across  (where 
the  inter-detector  angle  of  the  insect  is  1-1 .5°)  (Barnett  et  al  2007).  While  signaling  local  motion  of 
small  features,  these  small-field  STMDs  (SF-STMDs)  still  encapsulate  inputs  corresponding  to  a 
local  pool  of  dozens  of  adjacent  input  ommatidia,  so  could  easily  represent  the  output  of  an 
asymmetric,  non-linear  integration  of  local  ESTMDs  -  i.e.  a  2nd  order  motion  detector  network. 

Consistent  with  this  notion,  a  2nd  order  operation  was  recently  proposed  following  our  recent  finding 
(reported  in  the  early  stages  of  the  current  project)  that  the  response  time-course  in  the  large-field 
dragonfly  STMD  neuron  CSTMD1  to  small  target  stimuli  builds  to  a  maximum  over  several  hundred 
milliseconds  (Nordstrom  et  al  2010).  It  does  so,  however,  only  for  targets  that  successively  track 
across  tens  of  degrees  of  visual  angle,  and  thus  large  numbers  of  local  ommatidia,  as  quantified  in 
our  earlier  reporting  periods.  A  second-order  motion  detector  network  might  not  only  confer  direction 
selectivity,  but  could  potentially  enhance  target  detection  by  taking  advantage  of  a  distinguishing 
characteristic  feature  of  natural  target  motion:  true  targets  tend  to  move  along  continuous  paths, 
even  if  they  change  direction  or  vary  in  contrast  as  they  move  across  the  background.  A  response  in 
one  local  motion  detector  should  be  well  correlated  with  an  appropriately  delayed  response  in 
neighbouring  detectors  (i.e.  matching  the  target  velocity).  Noise,  on  the  other  hand  (including 
spurious  feature  motion  of  the  background,  such  as  foliage  moving  with  wind),  would  be  local  and 
inconsistent,  and  thus  less  likely  to  persist  along  continuous  trajectories.  A  second-order  system 
would  thus  enhance  rejection  of  feature  motion  not  correlated  across  multiple  local  adjacent  input 
detectors,  permitting  amplification  to  enhance  robustness  whilst  maintaining  underlying  selectivity  to 
stimuli  on  the  spatial  scale  of  single  ommatidia  of  the  eye. 

To  test  this,  we  further  developed  several  variants  of  computational  models  that  capture  the  key 
properties  of  direction-selective  SF-STMD  neurons  by  cascading  partially  rectified  ESTMD  stages 
with  HR-type  EMDs,  and  vice  versa.  We  then  examined  the  key  response  tuning  and  predictions  of 
these  second  order  systems  for  simple  stimuli.  We  developed  three  versions  of  the  target-detection 
model  each  including  identical  ‘biomimetic’  front-end  processing.  These  computational  models 
simulated  insect  optics  (blurring  and  hexagonal  sampling)  and  early  visual  processing  (dynamic 
bandpass  filtering)  as  described  in  the  Appendix.  These  3  model  variants  comprised: 
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(1 )  The  first  variant  was  simply  our  ESTMD  model  as  described  (and  applied)  above  (i.e.  in  Figure 
ID).  This  includes  strong  surround  antagonism  and  correlates  a  delayed  OFF  channel  with  an 
undelayed  ON  channel. 

(2)  A  second  variant  2nd  order  ‘hybrid’  EMD-ESTMD  model  (Figure  8A).  This  implements  a  front 
end  based  on  a  direction-selective  2-detector  EMD  (as  in  Fig.  IB)  that  separates  ON  and  OFF 
channels  and  then  subsequently  correlates  these  ‘like’  channels  in  separate  multiplication 
stages.  This  EMD  is  based  on  the  recent  findings  from  genetic  modification  of  fruit-fly  vision 
(Eichner  et  al.,  201 1 ,  Joesch  et  al.  201 3)  and  is  currently  regarded  as  the  ‘state  of  the  art'  as  a 
biomimetic  model  for  the  FIR-EMD  as  observed  in  insects.  In  our  hybrid  EMD-ESTMD  model 
(Figure  8A)  each  ‘ON-ON’  and  OFF-OFF  motion  signal  at  the  output  of  the  2  EMDs  then  serves 
as  input  to  a  2nd  order  ESTMD  stage  which  correlates  the  delayed  signal  from  the  OFF  EMD 
with  the  undelayed  ON  EMD. 

(3)  A  third  variant  2nd  order  hypdrid  (ESTMD-EMD)  variant  (Figure  8B)  implements  these  same 
operations  in  reverse  order,  i.e.  the  model  first  computes  the  ‘matched  target’  filter  with  an 
ESTMD  (as  in  Figure  1  b),  before  correlating  these  as  inputs  to  a  2nd  order  2-detector  EMD  (as  in 
Figure  1 B). 


Figure  8:  A.  The  cascaded  EMD-ESTMD  model  uses  spatiotemporally  correlated  ON  and 
OFF  channels  to  serve  as  inputs  to  an  ESTMD  model.  B.  The  cascaded  ESTMD-EMD  model 
adds  directionality  to  target  detection  by  correlating  the  output  of  two  non-directional  ESTMD 
units. 

Figure  9  shows  that  both  hybrid  models  lead  to  a  direction-selective  output.  The  figure  shows  the 
percentage  strength  of  directionality,  defined  here  relative  to  the  sum  of  the  response  to  targets 
moving  in  in  two  opposite  directions.  The  target  stimulus  subtended  an  angle  of  1 .25°  x  2°  and 
moved  at  a  speed  of  457s.  As  can  be  seen  the  ESTMD  responds  equally  to  either  target  direction 
and  is  in  effect  a  local  flicker  detector,  matched  to  the  spatiotemporal  profile  of  a  moving  target.  By 
cascading  a  2  detector  EMD  either  before  (EMD-ESTMD)  or  after  (ESTMD-EMD)  the  matched  filter 
induces  strong  directionality  to  the  model  responses,  with  responses  in  the  ‘preferred’  direction 
comprising  almost  100%  of  the  overall  response  to  both. 
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Figure  9:  Results  showing  that  the  ESTMD  model  responds  equally  to  target  motion  in  either 
direction  (1.25°  x  2°  at  457s),  while  addition  of  a  2  detector  EMD  cascaded  either  before 
(EMD-ESTMD,  Fig  8A)  or  after  (ESTMD-EMD,  Fig  8B)  the  matched  filter  induces  strong 
directionality  in  the  response  to  a  moving  target. 

Although  our  EMD  stage  incorporates  a  mirror  symmetric  subtraction  unit,  which  in  classical 
FIR-EMDs  leads  to  a  full  direction  opponent  response  (i.e.  an  inhibition  or  opposite  sign  response  to 
ant-preferred  direction  motion),  neither  of  the  hybrid  2nd  order  model  variants  produced  negative 
(opponent)  responses.  This  weak  directionality  mimics  a  property  observed  in  biological  (small-field) 
SF-STMD  neurons  (Barnett  et  al  2007).  In  this  respect,  such  neurons  contrasted  with  the  strong 
directional  opponency  observed  in  wide-field  optic  flow  neurons  that  served  as  the  primary 
inspiration  for  the  HR-EMD  model. 


Figure  10.  Size  tuning  of  the  three  model  variants  at  two  velocities  457s  (closed  circles)  and 
907s  (open  circles)  shows  tuning  curves  similar  to  those  observed  in  physiology.  The 
EMD-ESTMD  model  produces  a  tuning  curve  shifted  to  the  right  in  comparison  to  the  other 
models.  The  ESTMD-EMD  model  has  a  similar  optimum  to  the  ESTMD,  however  is  more 
sharply  tuned. 


To  examine  whether  model  variants  produce  similar  characteristics  to  those  observed  in 
physiological  STMD  experiments,  we  then  examined  the  target  size  tuning  of  the  model  variants  by 
simulating  target  motion  of  varying  width  (i.e.  their  spatial  direction  in  the  direction  of  travel)  and  at 
two  different  velocities  (Figure  10).  All  3  model  variants  produce  similar  response  characteristics, 
displaying  a  dependence  on  both  image  velocity  and  target  width.  The  EMD-ESTMD  size-tuning 
curve  is  sharper  and  shifted  to  the  right  (i.e.  less  selective  for  very  small  targets)  than  with  the  other 
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models.  The  ESTMD-EMD  tuning  curve  exhibits  a  similar  optimum  to  the  ESTMD  model,  however  is 
also  more  sharply  tuned.  At  a  lower,  less  optimal  velocity  (457s),  curves  are  slightly  left-shifted 
compared  to  a  velocity  of  907s,  indicating  the  confounded  relationship  between  the  velocity/width 
profile  of  a  moving  target. 


Figure  11:  Neuron  and  Model  output  in  response  to  moving  stimuli  of  varying  contrast.  A. 
Compares  the  contrast  sensitivity  function  (CSF)  for  responses  to  sinusoidal  grating  patterns 
of  optimum  frequency  in  FIS  cells,  a  class  of  neurons  in  the  fly  that  are  believed  to  collate  the 
oputput  from  large  arrays  of  classical  FIR-EMDs,  with  those  of  CSTMD1  (a  dragonfly  STMD 
neuron  selective  for  small  targets)  to  optimal  target  stimuli.  Because  the  wide-field  neuron 
integrates  motion  from  many  local  EMDs,  its  sensitivity  to  low  contrast  motion  is  higher  than 
the  STMD  (hence  our  selection  of  a  Log  axis).  Flowever  once  above  threshold,  the  STMD 
curve  rises  more  steeply.  B.  Shows  sensitivity  of  the  3  model  variants  to  Weber  contrast 
( Itarger  I  background )/ 1  background  >  a  measure  appropriate  for  small  targets  against  a  uniform 
background.  Although  all  3  variants  show  an  expansive  response  to  contrast,  the 
EMD-ESTMD  and  ESTMD-EMD  models  exhibit  similar  CSFs  with  a  higher-order 
dependence  on  contrast  (i.e.  steeper  rise)  than  the  ESTMD,  due  to  the  2nd  order  (cascaded) 
multiplicative  correlations.  C.  By  applying  an  arbitrary  gain  to  each  of  the  models  (to  match 
the  initial  expansive  ‘threshold’  for  model  response)  as  well  as  including  a  saturating 
non-linearity  (a  tanh  function),  we  show  that  the  cascaded  EMD/ESTMD  variants  might 
explain  the  smaller  operating  range  for  STMDs  with  respect  to  contrast. 

A  key  characteristic  of  the  FIR  EMD  is  that  output  in  response  to  increased  stimulus  contrast  is 
expansive,  due  to  the  super-linear  interaction  between  the  inputs  (usually  modeled  as  multiplication). 
In  biological  systems,  this  expansive  nonlinearity  is  inherently  bounded  by  the  saturation  limits  of 
synaptic  signaling  and  biochemistry  within  which  such  super-linear  operations  must  be  implemented. 
Flence  typical  neurons  taking  their  inputs  from  FIR-type  EMDs,  such  as  the  HS  neurons  of  flies, 
show  contrast-response  functions  that  are  initially  super-linear  before  becoming  sub-linear  as 
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saturation  begins  to  dominate  the  response  (Figure  11  A).  If  we  model  the  ESTMD  with  no  such 
saturation  on  its  outputs  (Figure  1 1 B),  it  displays  an  identical  expansive  characteristic,  due  to  the 
multiplication  stage.  The  EMD-ESTMD  and  ESTMD-EMD  show  very  similar  responses  to  increasing 
target  contrast.  Flowever,  as  expected  from  cascading  a  2nd  order  multiplication,  the  rate  of 
expansion  of  both  is  even  higher  order  (i.e.  4thorder)  than  in  the  ESTMD. 

In  this  respect,  these  cascaded  EMD/ESTMD  models  might  provide  a  simple  explanation  for  the 
pronounced  difference  in  shape  of  contrast  sensitivity  functions  for  insect  STMD  neurons  compared 
with  those  of  wide-field  optic  flow  neurons  which  give  responses  well  explained  by  HR-EMDs 
(Figure  1 1  A,  B).  Whereas  in  Figure  1 1B  we  normalize  model  outputs  to  their  own  maximum  (at 
contrast  1 .0),  in  order  to  compare  this  expansive  nonlinearity  across  the  model  variants  on  a  linear 
contrast  scale,  we  match  the  initial  expansion  with  arbitrary  gains  while  also  including  a  saturating 
nonlinearity  (hyperbolic  tangent),  a  function  that  mimics  the  soft-saturation  seen  in  typical 
physiological  motion  detecting  neurons  (Figure  1 1 C).  The  resulting  curves  (Figure  11)  suggest  that, 
once  we  account  for  response  saturation,  a  2nd  order  motion  detection  operation  could  explain  the 
hitherto  unexplained  steeper  dependence  of  contrast  sensitivity  of  STMDs  compared  with  that  of 
wide-field  motion  detectors  (Figure  11  A). 

4.4  Conclusions 

We  have  shown  that  several  alternative  correlation-based  models  (including  the  ESTMD)  can 
explain  a  number  of  basic  response  tuning  properties  of  insect  STMD  neurons,  such  as  their  size 
and  velocity  tuning.  Other  characteristics,  such  as  selectivity  for  contrast  polarity  (e.g.  dark)  and 
their  ability  to  discriminate  targets  in  clutter  without  relative  motion  cues,  can  presently  only  be 
explained  by  the  ESTMD  model.  Flowever,  some  STMDs  also  exhibit  characteristics  that  the 
ESTMD  cannot  account  for,  including  directionality,  facilitation  and  strongly  expansive  contrast 
sensitivity.  By  elaborating  the  ESTMD  model  with  a  cascaded  EMD,  we  induce  directionality  (Figure 
9)  maintain  the  size  and  velocity  tuning  intrinsic  to  our  ESTMD  model  (Figure  10)  and  can  also 
explain  the  highly  supra-linear  response  to  target  contrast  that  distinguishes  many  insect  STMD 
neurons  (Figure  1 1 ).  A  2nd  order  motion  detector  network  similar  to  those  we  implement  was 
previously  proposed  to  explain  the  ‘facilitated’  responses  observed  in  some  STMDs  when  targets 
move  along  continuous  trajectories  (Nordstrom  et  al  201 1 ,  Dunbier  et  al  2012). 

One  of  our  objectives  was  to  determine  whether  one  or  other  of  the  cascaded  model  variants 
matched  observed  physiological  results  and  whether  in  fact  one  variant  could  be  ‘ruled  out’  due  to 
an  inconsistency.  As  both  models  match  the  data  we  have  supported  the  existence  of  a  2nd  order 
network,  however,  more  physiological  experiments  and  modeling  will  be  required  to  elucidate  the 
underlying  architecture.  In  future  work  it  will  be  interesting  to  stimulate  both  the  neurons  and  our 
model  variants  with  non-Fourier  motion  and  see  whether  this  can  be  used  to  determine  which 
architecture  is  the  more  likely  candidate  to  underlie  STMD  processing.  In  future  experiments  we  will 
present  also  moving  targets  in  cluttered  environments  to  our  model  variants  and  determine  whether 
this  facilitation  breaks  or  enhances  target  discrimination. 

Overall  achievements  and  future  work: 

In  the  last  decade  the  Principal  Investigators  on  this  project  have  established  the  underlying  neural 
machinery  of  the  insect  STMD  system  as  an  important  new  model  system  for  computational 
neuroscience  and  for  bio-inspired  models  of  target  tracking.  With  sponsorship  from  the  US  Air  Force 
Office  of  Scientific  Research  (Contracts  F49620-01-C-0030,  FA9550-04-1-0283,  and  Grants 
FA9550-09-1  -01 1 6  and  FA2386-1 0-1  -41 1 4),  our  research  has  allowed  us  to  characterize  and  model 
key  physiological  properties  that  underlie  the  impressive  visual  abilities  of  insects  such  as  predatory 
dragonflies  to  discriminate  and  track  small  moving  targets  against  cluttered  backgrounds,  despite 
poor  optical  resolution  (low  pixel  count).  In  the  process,  we  have  established  a  foundational 
literature  on  this  topic,  including  a  number  of  high-profile  publications  in  top  international  journals 
from  this  current  grant  that  received  substantial  media  coverage. 

As  the  final  project  in  the  series  drew  to  a  close,  we  obtained  several  significant  electrophysiological 
results  that  significantly  impact  our  understanding  of  the  processing  that  underlies  small  target 
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detection.  These  either  confirm  hypothesized  mechanisms  or  fill  in  important  gaps  in  our 
understanding,  and  the  resulting  models  form  an  ideal  platform  for  future  translational  and  robotics 
efforts.  They  culmination  of  our  modeling  effort  was  a  computational  model  that  was  highly 
speculative  in  the  early  stages  of  the  project,  but  which  is  now  on  the  verge  of  a  mature,  explanatory, 
and  physiologically-grounded  status. 

We  remain  the  world's  leading  experts  on  this  facet  of  insect  vision,  which  has  heretofore  received 
little  attention.  Yet  feature  tracking  may  be  at  least  as  important  for  visually  guided  behavior  as  the 
better  studied  and  more  famous  insect  wide-field  motion  analysis  (optic  flow)  system.  Furthermore, 
just  as  understanding  insect  optic  flow  analysis  is  expected  to  have  a  significant  impact  on  visual 
guidance  in  autonomous  microsystems,  so  might  small  target  detection  for  military  applications 
such  as  seekers  and  the  detection  of  aircraft  for  collision  avoidance  or  surveillance  purposes. 
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4.6  Appendix  1:  ESTMD  Model  description 
Model  overview 

Models  were  implemented  in  Matlab  (Mathworks,  Natick,  USA).  An  overview  of  early  visual 
processing  and  the  ESTMD  model  is  shown  in  Figure  12.  The  EMD  model  and  combination  model 
variants  are  shown  in  Figure  1&  Figure  8. 


A.  Early  Visual  Processing 
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Figure  12:  A.  Overview  of  early  visual  processing.  B.  The  main  elements  of  the  elementary 
small  target  motion  detector  (ESTMD).  Inputs  are  spatially  blurred  to  represent  fly  optics.  We 
retain  only  the  green  channel  of  the  RGB  image,  to  represent  spectral  sensitivity  of  the  motion 
pathway.  A  complex  photoreceptor  model  implements  dynamic  filtering  characteristics  and 
adaptive  feedbacks  which  allow  for  the  encoding  of  vast  luminance  conditions.  LMCs  are 
modeled  as  dynamic  spatiotemporal  high-pass  filters  (relaxed),  removing  redundant 
information.  The  model  implements  functionality  inspired  from  electrophysiological  recordings 
of  RTCs,  found  in  the  brain  of  the  fly.  This  includes  ON  and  OFF  channel  separation, 
independent  fast  temporal  adaptation  and  independent  channel  surround  antagonism.  Finally, 
the  delayed  OFF  channel  is  recombined  with  the  undelayed  ON  channel  for  dark  target 
sensitivity.  The  final  output  reveals  enhanced  small  target  discrimination  as  seen  in 
physiological  STMDs. 


Photoreceptor  responses  were  based  on  a  biomimetic  model  with  parameters  and  elaborations 
derived  via  electrophysiological  results  from  Eristalis  tenax.  The  spatiotemporal  dynamics  of  the  first 
order  interneurons,  the  LMCs,  have  previously  been  established  for  blowfly  (Calliphora  vicina )  and 
hoverfly,  (Eristalis  tenax).  Model  outputs  were  matched  to  STMD  neurons  in  Eristalis  tenax.  We 
then  modeled  an  array  of  ESTMD  (elementary  STMD)  subunits  that  could  be  spatially  pooled  to 
form  the  position  invariant  receptive  field,  as  seen  in  the  physiological  STMD.  The  model  output 
define  values  in  a  three  dimensional  (2D+t)  space. 
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Early  Visual  Processing 

We  used  the  green  channel  in  our  8-bit  input  imagery  to  simulate  ‘green-blue’  spectral  sensitivity  in 
the  fly  visual  system.  We  applied  a  Gaussian  low-pass  filter  (full  width  at  half  maximum  1 .4°)  to 
emulate  spatial  blur  of  the  optics.  Input  images  were  spatially  sampled  at  1  °  in  a  hexagonal  manner. 
Photoreceptors  incorporated  variable  gain  control,  saturating  nonlinearities  and  dynamic  low-pass 
filtering,  with  cutoff  frequencies  dependent  on  adaptation  state  (ranging  from  20  to  1 00  Hz).  This 
was  followed  by  two  divisive,  delayed  feedbacks  (one  linear,  one  exponential)  representing  short 
and  longer  term  adaptations  (t  =  23  ms,  t  =  12.4  s).  Finally,  a  compressive,  saturating  nonlinearity  is 
implemented  by  a  Naka-Rushton  transform.  The  LMC  implements  spatiotemporal  high-pass  filtering, 
altering  its  filtering  characteristics  dependent  on  visual  conditions.  In  the  dark  adaptation  state,  the 
LMC  is  more  integrative  with  longer  sustaining  temporal  components.  As  overall  luminance 
conditions  increase,  the  LMC  becomes  more  transient  and  high-pass  in  nature,  both  in  space  and 
time.  This  spatial  interaction  (center-surround  antagonism)  was  modeled  in  a  feed  forward  manner 
with  surround  (nearest  neighbor)  photoreceptor  signals  summed,  and  temporally  delayed  (t=16  ms), 
before  subtractively  inhibiting  the  central  LMC  (strength  ranging  from  0-30%,  dependent  on 
adaptation  state).  The  LMC  temporal  dynamics  were  modeled  with  relaxed,  variable  high-pass 
filtering  (lower  corner  frequency  ranging  from  0  to  8  Hz)  which  incorporated  a  small  DC  component 
(0-10%).  Following,  a  saturating  nonlinearity  with  a  hyperbolic  tangent  function,  ensured  the  LMC 
response  was  limited  to  a  predictable  output  range. 

Elementary  Small  Target  Motion  Detector  (ESTMD) 

The  main  processing  of  the  ESTMD  is  based  on  the  Rectifying  Transient  Cell  (RTC).  Briefly,  the 
RTC  creates  transient  ‘on’  and  'off  phases  (from  the  LMC  high-pass  filtering),  separated  via  a 
further  temporal  high  pass  filter  (t  =  40ms)  and  then  half-wave  rectification  into  independent  ON 
and  OFF  channels.  Each  of  the  channels  is  temporally  processed  through  a  fast  adaptive 
mechanism.  An  adaptation  state  is  determined  by  a  nonlinear  filter,  which  approximates  cellular  ‘fast 
depolarization  and  slow  repolarization’  responses.  This  low-pass  filter  switches  its  time  constant 
dependent  on  whether  the  input  is  increasing  or  decreasing  (time  constants  are  ‘fast’  (t  =  3  ms) 
when  channel  input  is  increasing  and  ‘slow’  (t  =  70  ms)  when  decreasing).  This  adaptation  state 
subtractively  inhibits  the  unaltered  ‘pass-through’  signal.  The  result  of  this  complex,  nonlinear 
filtering  is  the  signaling  of  ‘novel’  transient  contrast  changes  (of  the  particular  channel  phase,  ‘on’  or 
‘off’)  with  the  suppression  of  fluctuating  textural  variations.  As  well  as  this  temporal  antagonism,  the 
channels  also  exhibit  spatial  antagonism  with  ON  surround  channels  subtractively  inhibiting  the  ON 
centre  channel,  and  similarly  with  the  OFF  channels.  The  resultant  signal  was  then  half-wave 
rectified,  so  that  the  surround  does  not  inhibit  the  centre  below  a  zero  value  (a  nonlinearity  seen  in 
some  spiking  neurons).  ON  and  OFF  channels  are  recombined  via  multiplication  [6].  Both  dark  and 
light  target  sensitivity  is  possible  by  delaying  and  recombining  the  relevant  contrast  polarity.  For 
these  experiments  we  delayed  the  OFF  channel  using  a  1st-order  low-pass  filter  (t=  25  ms)  and 
multiplying  this  by  the  undelayed  ON  channel.  This  processing  provides  a  template  for  the 
characteristic  temporal  ‘signature’  of  a  small  moving  target. 

Elementary  Motion  Detector  (EMD) 

The  2-Detector  Elementary  Motion  Detector  (EMD)  as  described  by  Eichner  et  al.(  201 1 )  and 
Joesch  et  al.  (201 3)  was  implemented  as  described  above  (Figure  1 ).  Channels  were  separated  into 
ON  and  OFF  channels  via  half-wave  rectification  following  high  pass  filtering  (t=100  ms).  The  delay 
arm  was  modeled  with  a  low-pass  filter  (t=  25  ms). 
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